Testing right now.

Yesterday, with this command line (equivalent to pre-PR 728)

llama-perplexity -m X:\text-generation-webui\models\zai-org_GLM-4.5-Air-bf16-iMat-IQ4_XS_L-00001-of-00005.gguf -f wiki.test.raw --parallel 1 -ngl 150 -b 512 -mg 0 -ts 18,18,12 --no-mmap -no-fmoe -no-mmad -cuda fusion=1,offload-batch-size=1,mmq-id-size=0 -c 512 --override-kv glm4moe.expert_used_count=int:7

I obtained: Final estimate: PPL = 4.6508 +/- 0.02854

Today, with this command line, activating MMQ_ID and the proposed fix :

llama-perplexity -m X:\text-generation-webui\models\zai-org_GLM-4.5-Air-bf16-iMat-IQ4_XS_L-00001-of-00005.gguf -f wiki.test.raw --parallel 1 -ngl 150 -b 512 -mg 0 -ts 18,18,12 --no-mmap -no-fmoe -no-mmad -cuda fusion=1,offload-batch-size=1,mmq-id-size=1000 -c 512 --override-kv glm4moe.expert_used_count=int:7

I obtained: PPL over 565 chunks for n_ctx=512 = 4.6520 +/- 0.02855

The final PPL bump is 1/20th of what it was, the variance on individual values are 1/10th of what they were, and we're back in the realm of an "acceptable" trade-off for the performance boost.

With FMOE activated :

Final estimate: PPL over 565 chunks for n_ctx=512 = 4.6520 +/- 0.02855 (no variance, I had one yesterday without MMQ_ID).

With MMAD activated :

Final estimate: PPL over 565 chunks for n_ctx=512 = 4.6513 +/- 0.02855 (same final PPL than without MMQ_ID with the same options, but the intermediary variances are higher, especially in the beginning of the test (+0.02 PPL, very much like before the proposed MMQ_ID fix)).

MMAD repeatedly causes variances no matter which other options are chosen. I'll deactivate it for now on my side, and also, I got a PP speed drop of 1.5% during my perplexity test with MMAD on. I would suggest you to review PR #858 again.

but the intermediary variances are higher, especially in the beginning of the test

You mean they differ from one PPL run to another PPL run with the same parameters, or they differ from the run without the MMAD fusion? If the latter, this is very normal. If the former, I cannot reproduce. There is a data race in the add + fused_rms_norm fusion, but this should be solved after #916 has been merged.

I'm sorry, I wondered afterwards if I was ambiguous, and I was.

The latter, of course. With MMQ_ID fixed and activated, MMAD modifies quite a bit the intermediary values of the PPL test, even if we end up with a final PPL similar to no-mmad.

Again thank you for reporting this issue, it was undetected on the main repository for months. Fix here: ggml-org/llama.cpp#17089 .

Rounding up the number of SMs is only a complete fix for a subset of tensor shapes (they need to be sufficiently large); I use it for debugging but I would strongly advise you to take over the PR I linked since there is simply an index in the stream-k fixup that is being calculated correctly.